REMOTE-2111 (2/3): Add periodic checkpoint coordinator state machine by joeywangzr · Pull Request #14573 · warpdotdev/warp

joeywangzr · 2026-07-31T03:50:36Z

Part 2 of 3 in a stack splitting the periodic workspace-handoff checkpoint client work into reviewable chunks.

Stack

REMOTE-2111 (1/3): Add checkpoint upload/commit pipeline mechanics #14588 — checkpoint upload/commit mechanics (base: master)
This PR — periodic checkpoint coordinator state machine (base: REMOTE-2111 (1/3): Add checkpoint upload/commit pipeline mechanics #14588's branch)
REMOTE-2111 (3/3): Wire the periodic checkpoint coordinator into AgentDriver #14589 — wire the coordinator into AgentDriver (base: this branch)

Stacked on #14588; only the diff vs. that branch is new in this PR. Please review against #14588's branch, not master.

What this does

Adds the Idle -> Due -> InFlight -> Idle scheduling loop (with Finalizing -> Stopped reachable from any state) that drives periodic workspace-handoff checkpoints:

checkpoint_coordinator.rs: CheckpointCoordinatorHandle spawns the loop and exposes finalize() for graceful shutdown. The timer only ever moves Idle to Due; Due polls a safe-boundary predicate (conversation not mid-turn) before entering InFlight, which runs exactly one attempt via snapshot::run_checkpoint_from_declarations_file on a background task so a racing finalize() can bound how long it waits without ever stranding the attempt. finalize() starts at most one more attempt (skipped below the gather/upload timeout floor) or awaits an already-in-flight one — never both — bounded end to end by its budget.
terminal/view.rs: add TerminalView::ai_action_model(), which is_safe_boundary uses to check for unfinished actions on the conversation.

Note on #![allow(dead_code)]: this module has no production caller yet — AgentDriver wires CheckpointCoordinatorHandle::new into its spawn/finalize lifecycle in #14589 — so it's marked at the module level with an explanatory comment. Becomes genuinely reachable once the stack is applied in full.

Validation

cargo clippy --all-targets --tests -- -D warnings, cargo fmt --check, and the full coordinator test suite (13 tests covering jitter bounds, finalize-below/above-floor from both Idle and InFlight, safe-boundary deferral and retry, interval timing, and orphan-safety) all pass on this branch.

Co-Authored-By: Oz oz-agent@warp.dev

oz-for-oss · 2026-07-31T03:50:49Z

@joeywangzr

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR adds client-side periodic workspace-handoff checkpoints, including a coordinator, checkpoint-mode upload/commit requests, and tests around the new pipeline.

Concerns

End-of-run finalization passes only the upload timeout to the coordinator, but idle/due finalization requires budget greater than script timeout plus upload timeout, so default runs can skip the final checkpoint entirely.
The coordinator-owned finalization path bypasses the legacy declarations-writer flush, which can omit recently queued file declarations from the selected checkpoint.
The checkpoint commit path allows Skipped upload entries, including missing upload targets for required blobs, to be committed as a selected checkpoint.

Verdict

Found: 0 critical, 3 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss · 2026-07-31T03:56:46Z

+        // selected checkpoint. This replaces the legacy one-shot upload below so
+        // there is exactly one end-of-run snapshot path, not two.
+        if let Some(coordinator) = checkpoint_coordinator {
+            coordinator.finalize(upload_timeout).await;


⚠️ [IMPORTANT] Passing only upload_timeout means finalize_with_new_attempt will skip whenever the coordinator is Idle/Due, because it requires the remaining budget to be greater than script_timeout + upload_timeout; with the defaults, runs that finish before the first periodic checkpoint replace the legacy end-of-run snapshot with no checkpoint at all.

oz-for-oss · 2026-07-31T03:56:46Z

+    script_timeout: Duration,
+    upload_timeout: Duration,
+) -> CheckpointResult {
+    snapshot::run_declarations_script(&working_dir, &task_id, script_timeout).await;


⚠️ [IMPORTANT] This regenerates declarations without first flushing snapshot_file_writer; the legacy path drains queued file declarations before running the script, so recent orphan-file edits can remain queued and be omitted from the committed checkpoint.

oz-for-oss · 2026-07-31T03:56:46Z

+    if outcome
+        .entries
+        .iter()
+        .any(|e| e.status == EntryStatus::Failed)


⚠️ [IMPORTANT] Skipped also represents requested blobs that received no upload target, so committing whenever there are no Failed entries can select a partial checkpoint that is missing required file contents; distinguish cap-skipped entries from missing-target skips or treat non-cap skips as commit-blocking.

joeywangzr · 2026-07-31T14:16:51Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

joeywangzr · 2026-07-31T15:45:02Z

Pushed a follow-up commit to this branch from a review pass (e3bc04f), and rebased onto the updated mechanics branch. Summary:

Blocking — the finalize floor was unreachable, so the end-of-run checkpoint never ran.
finalize_with_new_attempt gates on remaining > script_timeout + upload_timeout, but PR #14589 passes upload_timeout alone as the budget. That comparison is false for every possible configuration (defaults: 120s > 60s + 120s; even with --snapshot-script-timeout 0 it's 120s - ε > 120s). Since the coordinator returns past the legacy one-shot upload once active, enabling the flag meant no end-of-run snapshot at all — runs shorter than one interval uploaded nothing, longer runs lost everything since the last periodic tick.

Added finalize_budget(script_timeout, upload_timeout) so the floor has exactly one owner and callers can't drift from it; #14589 now uses it. Also downgraded the skip log to WARN, since reaching it means the end-of-run checkpoint was dropped. coordinator_finalize_from_idle_skips_attempt_below_floor was the only test covering this gate, and it asserted the skip — I kept it and added the positive case.

Blocking — the loop could never terminate once the driver was gone.
is_safe_boundary mapped Err(ModelDropped) to "safe", so a dropped AgentDriver left the coordinator gathering and uploading the whole workspace every interval forever. Nothing else stops the loop, and run_snapshot_upload returns without finalizing under exactly that condition (spawner round trip fails, or OzHandoff toggled off). Replaced the boolean with a Boundary tri-state so DriverGone returns from coordinator_loop.

Blocked conversations never checkpointed. Blocked is itself backed by a pending action, so it fell through to has_unfinished_actions_for_conversation and reported busy forever — a run parked on user approval (often hours) polled every 2s and never checkpointed, though nothing was mutating the workspace. Blocked and is_done are now quiescent alongside WaitingForEvents. Also added MAX_BOUNDARY_DEFERRAL (10m): without a cap, any turn longer than the interval starves the feature in exactly the long-running case it exists for. A slightly-inconsistent checkpoint beats none, since the previous generation is only replaced on success.

Boundary polling now stays responsive to finalize. The check was awaited outside select!; it's a round trip through the model task queue, so a stall could wedge shutdown behind an unbounded await.

Declarations writer is flushed before every attempt. run_snapshot_upload does this on the legacy path specifically so no driver-side file append is in flight when the bash script starts appending to the same JSONL; the coordinator didn't, so checkpoints could miss the agent's latest edits and race the script.

Also gave finalize's outer ack timeout slack over the inner deadline — bounding both on exactly budget meant the "defense-in-depth" bound would routinely preempt an attempt still legitimately running.

Five tests added (derived budget clears the floor; finalize during boundary polling still attempts; gone driver stops the loop; loop stops after finalize; failed attempt retries next interval). All 16 coordinator tests pass locally, plus clippy -D warnings and fmt.

Two things I left alone but flagged: periodic attempts route every failure through report_error!, which at a 5-minute cadence over a multi-hour run will flood Sentry if the endpoint is unhealthy; and gather_failed/read_failed entries still commit, so a transient git diff failure can replace a complete checkpoint with one missing that repo's patch.

Adds the Idle -> Due -> InFlight -> Idle scheduling loop (with Finalizing -> Stopped reachable from any state) that will drive periodic workspace-handoff checkpoints, per REMOTE-2111 Phase 3. - checkpoint_coordinator.rs: CheckpointCoordinatorHandle spawns the loop and exposes finalize() for graceful shutdown. The timer only ever moves Idle to Due; Due polls a safe-boundary predicate (conversation not mid-turn) before entering InFlight, which runs exactly one attempt via snapshot::run_checkpoint_from_declarations_file on a background task so a racing finalize() can bound how long it waits without ever stranding the attempt. finalize() starts at most one more attempt (skipped below the gather/upload timeout floor) or awaits an already-in-flight one -- never both -- bounded end to end by its budget. - terminal/view.rs: add TerminalView::ai_action_model(), which is_safe_boundary uses to check for unfinished actions on the conversation. This module has no production caller yet -- AgentDriver wires CheckpointCoordinatorHandle::new into its spawn/finalize lifecycle in a follow-up, stacked PR -- so it is marked allow(dead_code) at the module level with a comment explaining why; removable once that PR merges on top of this one. Co-Authored-By: Oz <oz-agent@warp.dev>

Review follow-ups on the periodic checkpoint coordinator. 1. Expose `finalize_budget` so the finalize floor is reachable. `finalize_with_new_attempt` gates on `remaining > script_timeout + upload_timeout`. The wiring PR passes `upload_timeout` alone as the budget, so that comparison is false for *every* configuration (defaults: 120s > 60s + 120s), and because the coordinator owns the whole end-of-run path once enabled, the result is no end-of-run snapshot at all. Add `finalize_budget(script_timeout, upload_timeout)` and document that callers must derive the budget from it, so `AgentDriver` cannot drift out of sync with the floor. Downgrade the skip log to WARN, since reaching it means the end-of-run checkpoint was dropped. Also give `finalize`'s outer ack timeout slack over the inner deadline. Bounding both on exactly `budget` meant the "defense-in-depth" bound would routinely preempt an attempt the coordinator was legitimately still finishing, rather than only catching a bug. 2. Stop the loop when the driver is gone. `is_safe_boundary` mapped `Err(ModelDropped)` to "safe", so a dropped `AgentDriver` left the coordinator gathering and uploading the whole workspace every interval forever. Nothing else stops the loop, and `run_snapshot_upload` returns without finalizing under exactly the same condition (spawner round trip fails, or `OzHandoff` is off). Replace the boolean predicate with a `Boundary` tri-state so `DriverGone` returns from `coordinator_loop` instead of being conflated with `Safe`. 3. Checkpoint blocked conversations, and cap boundary deferral. `Blocked` is backed by a pending action, so it fell through to `has_unfinished_actions_for_conversation` and reported busy forever: a run parked on user approval (often hours) polled every 2s and never checkpointed, though nothing was mutating the workspace. Treat `Blocked` and `is_done` as quiescent alongside `WaitingForEvents`. Add `MAX_BOUNDARY_DEFERRAL` (10m) so a long turn can no longer starve the feature entirely. A slightly-inconsistent checkpoint beats none, since the previous committed generation is only replaced on success. 4. Keep boundary polling responsive to finalize. The boundary check was awaited outside `select!`, so a stalled model task queue could wedge shutdown behind an unbounded await. `wait_for_safe_boundary` now races it against the finalize channel. 5. Flush the declarations writer before every attempt. `AgentDriver::run_snapshot_upload` flushes pending driver-side `file` appends before running the declarations script, precisely so no write is in flight when the bash script starts appending. The coordinator did not, so both periodic and final checkpoints could miss the agent's most recent edits and race the script on the shared append-only JSONL. Thread an optional `DeclarationsWriterHandle` through and flush it in `run_one_attempt`. 6. Correct `DEFAULT_CHECKPOINT_INTERVAL`'s doc: the cadence is measured from attempt completion, not attempt start. Tests: the derived budget clears the floor (and the old pairing provably cannot); finalize during boundary polling still runs an attempt; a gone driver stops the loop without attempting; the loop stops after finalize; and a failed attempt returns to Idle and retries on the next tick. Co-Authored-By: Oz <oz-agent@warp.dev>

cla-bot Bot added the cla-signed label Jul 31, 2026

oz-for-oss Bot reviewed Jul 31, 2026

View reviewed changes

joeywangzr changed the base branch from master to graphite-base/14573 July 31, 2026 14:16

joeywangzr force-pushed the jopee/remote-2111-checkpoint-coordinator branch from 944a150 to 2f4d085 Compare July 31, 2026 14:16

joeywangzr changed the base branch from graphite-base/14573 to jopee/remote-2111-checkpoint-mechanics July 31, 2026 14:16

This was referenced Jul 31, 2026

REMOTE-2111 (1/3): Add checkpoint upload/commit pipeline mechanics #14588

Open

REMOTE-2111 (3/3): Wire the periodic checkpoint coordinator into AgentDriver #14589

Open

joeywangzr changed the title ~~REMOTE-2111: client-side periodic workspace-handoff checkpoints~~ REMOTE-2111 (2/3): Add periodic checkpoint coordinator state machine Jul 31, 2026

joeywangzr force-pushed the jopee/remote-2111-checkpoint-coordinator branch from 2f4d085 to e3bc04f Compare July 31, 2026 15:44

joeywangzr force-pushed the jopee/remote-2111-checkpoint-coordinator branch from e3bc04f to f9880b7 Compare July 31, 2026 19:26

joeywangzr and others added 2 commits July 31, 2026 16:08

joeywangzr force-pushed the jopee/remote-2111-checkpoint-coordinator branch from f9880b7 to 6735cd6 Compare July 31, 2026 20:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

REMOTE-2111 (2/3): Add periodic checkpoint coordinator state machine - #14573

REMOTE-2111 (2/3): Add periodic checkpoint coordinator state machine#14573
joeywangzr wants to merge 2 commits into
jopee/remote-2111-checkpoint-mechanicsfrom
jopee/remote-2111-checkpoint-coordinator

joeywangzr commented Jul 31, 2026 •

edited

Loading

Uh oh!

oz-for-oss Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

oz-for-oss Bot left a comment

Uh oh!

oz-for-oss Bot Jul 31, 2026

Uh oh!

oz-for-oss Bot Jul 31, 2026

Uh oh!

oz-for-oss Bot Jul 31, 2026

Uh oh!

joeywangzr commented Jul 31, 2026

Uh oh!

joeywangzr commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

joeywangzr commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stack

What this does

Validation

Uh oh!

oz-for-oss Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

oz-for-oss Bot Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

oz-for-oss Bot Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

oz-for-oss Bot Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

joeywangzr commented Jul 31, 2026

Uh oh!

joeywangzr commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

joeywangzr commented Jul 31, 2026 •

edited

Loading

oz-for-oss Bot commented Jul 31, 2026 •

edited

Loading